DOCUMENT RESUME 



ED 475 318 

AUTHOR 

TITLE 

INSTITUTION 
SPONS AGENCY 
REPORT NO 
PUB DATE 
NOTE 

CONTRACT 
AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



EC 309 498 



Thompson, Sandra; Thurlow, Martha; Moore, Michael 

Using Computer-Based Tests with Students with Disabilities. 

NCEO Policy Directions. 

National Center on Educational Outcomes, Minneapolis, MN. 
Special Education Programs (ED/OSERS), Washington, DC. 

NCEO-15 

2003-01-00 

8p. 

H326G000001 

National Center on Educational Outcomes, University of 
Minnesota, 350 Elliott Hall, 75 East River Rd., Minneapolis, 
MN 55455 ($3.50). Tel: 612-624-8561; Fax: 612-624-0879; e- 
mail: nceo0umn.edu. For full. text: http://education.umn.edu/ 
NCEO. 

Guides - Non-Classroom (055) 

EDRS Price MFOl/PCOl Plus Postage . 

^Computer Assisted Testing; ^Disabilities ; Educational 
Testing; Elementary Secondary Education; Language Minorities; 
*Test Construction; *Test Format; ^Testing Accommodations; 
Testing Problems 



ABSTRACT 

This report presents factors to consider in the design of 
computer-based testing for all students, including students with disabilities 
and students with limited English proficiency. It also provides a process for 
the initial transformation of paper/pencil assessments to inclusive computer- 
based testing. Steps include: (1) assemble a group of experts to guide the 

transformation, including experts on assessment design, accessible Web 
design, universal design, and assistive technology; (2) decide how each 
accommodations will be incorporated into the computer-based test; (3) 
consider each accommodation or assessment feature in light of the constructs; 
(4) consider the feasibility of incorporating the accommodation into 
computer-based tests; arid (5) consider training implications for staff and 
students with- special attention being given to the computer literacy of 
students and their experience using features like screen readers. The report 
then provides the following examples of specific accommodations for students 
with disabilities and students with limited English proficiency: large print 
and magnification, audio presentation of instruction and test items, 
simplified instructions, writing in the test booklet, using a calculator, 
breaks and multiple test sessions, and individual or small group 
administration. The report closes with .a list of considerations in the 
transformation of accommodations from paper/pencil to computer-based tests. 
(Contains 10 references.) (CR) 
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Using Computer-based Tests with 
Students with Disabilities 




Computer-based testing has been 
called the "next frontier in testing" 
as educators, testing companies, 
and state departments quickly 
work to transform paper/ pencil 
tests into technology-based for- 
mats. These efforts have occurred 
in a variety of ways and for a 
variety of tests. Some educators 
have transferred all of their class- 
room quizzes and tests into a 
computer-based format. 

With the dramatic increase in the 
use of the Internet over the past 
few years, and the considerable 
potential of onhne learning, assess- 
ment will need to undergo a 
complete transformation to keep 
pace. Experts suggest that the 
Internet will be used to develop 
tests and present items through 
dynamic and interactive stimuli 
such as audio, video, and anima- 
tion. Given this momentum, it is 
not surprising that there is a trend 
toward investigating and incorpo- 
rating the Internet as the testing 
medium for statewide assessments. 

O 



Computer-based testing is viewed 
by many policymakers as a way to 
meet the requirements of the No 
Child Left Behind Act of 2001 
(NCLB). The need to produce 
itemized score analyses, disaggre- 
gation within each school and 
district by gender, racial and ethnic 
group, migrant status, English 
proficiency, disability, and income 
challenges states to create new and 
more efficient ways to administer, 
score, and report assessment 
results. 

There clearly are many opportuni- 
ties created when computer-based 
tests are used. These include more 
efficient test administrations, the 
availability of immediate results, 
and student preferences for this 
form of testing over paper and 
pencil tests. In addition, computer- 
based testing opens up the possi- 
bility for built-in accommodations, 
student selection of testing op- 
tions, and increased authenticity in 
items that are included. Other 
benefits have been identified as 
well, so there is considerable 
pressure to move toward com- 
puter-based testing. 



While computer-based testing may 
address the challenges of NCLB 
and has many other positive 
characteristics as well, it poten- 
tially creates other problems unless 
a thoughtful and systematic pro- 
cess is used to transfer existing 
paper/pencil assessments to 
computer-based assessments. Not 
only will poor design elements on 
the paper test transfer to the 
screen, but additional challenges 
may result that reduce the validity 
of the assessment results and 
possibly exclude some groups of 
students from assessment partici- 
pation. 

This Policy Directions presents 
factors to consider in the design of 
computer-based testing for all 
students, including students with 
disabilities and students with 
limited English proficiency. It also 
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provides a process for the initial 
transformation of paper/ pencil 
assessments to inclusive computer- 
based testing. 

A report to the National Gover- 
nors' Association sums up what 
we need to remember as computer- 
based testing grows across the 
United States and throughout the 
world: 

Do not forget why electronic 
assessment is desired. Electronic 
assessment will enable states to 
get test results to schools faster 
and, eventually, cheaper. It will 
help ensure assessment keeps 
pace with the tools that students 
are using for learning and with 
the ones that adults are increas- 
ingly using at work. The tech- 
nology will also help schools 
improve and better prepare 
students for the next grade, for 
postsecondary learning, and for 
the workforce. (Using Electronic 
Assessment to Measure Student 
Performance, 2002, p. 9) 




Despite the potential advantages 
offered by computer-based testing, 
there remain several challenges, 
especially in the transition from 
paper/pencil assessments. First of 
all, the use of technology cannot 
take the place of content mastery. 
No matter how well a test is 
designed, or what media are used 
for administration, students who 
have not had an opportunity to 
learn the material tested will 
perform poorly. Students need 
access to the information tested in 
order to have a fair chance at 
performing well. Researchers 
strongly caution that the use of a 
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computer, in and of itself, does not 
improve the overall quality of 
student writing. We continue to 
find significantly lower mean test 
scores for students with disabilities 
than for their peers without dis- 
abilities. The following are some 
challenges that must be overcome 
in order for computer-based 
testing to be effective. 

Issues of Equity and Skill in 

Computer Use 

Concerns continue to exist in the 
area of equity, where questions are 
asked about whether the required 
use of computers for important 
tests puts some students at a 
disadvantage because of lack of 
access, use, or familiarity. Concerns 
include unfamiliarity with answer- 
ing standardized test questions on 
a computer screen, using buttons 
to search for specific items, and 
indecision about whether to use 
traditional tools (e.g., hand held 
calculator) vs. computer-based 
tools. 

^ Added Challenges for Some 
Students 

Some research questions whether 
the medium of test presentation 
affects the comparability of the 
tasks students are being asked to 
complete. For example, (1) com- 
puter-based testing places more 
demands on certain skills such as 
typing, using multiple screens to 
recall a passage, mouse navigation, 
and the use of key combinations, 

(2) some people become more 
fatigued when reading text on a 
computer screen than on paper, (3) 
long passages may be more diffi- 
cult to read on a computer screen, 
and (4) the inability to see an entire 
problem on screen at one time is 
challenging. 
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[>Lack of Ability to Design 
Accessible Web Pages 

According to Web AIM, (Web 
Accessibility in Mind, an initiative 
of the Center for Persons with 
Disabilities at Utah State Univer- 
sity), there are 27.3 million people 
with disabilities who are limited in 
the ways they can use the Internet: 
"The saddest aspect of this fact is 
that the know-how and the tech- 
nology to overcome these limita- 
tions already exist, but they are 
greatly under-utilized, mostly 
because Web developers simply do 
not know enough about the issue 
to design pages that are accessible 
to people with disabilities." 




Inclusive Computer- 



The transformation of traditional 
paper/pencil tests to inclusive 
computer-based tests takes careful 
and thorough work that includes 
the collaborative expertise of many 
people. Five steps should be used 
to address these transformation 
issues (see Table 1). 



Step 1. Assemble a group of 
experts to guide the transforma- 
tion. Include experts on assess- 
ment design, accessible Web 
design, universal design, and 
assistive technology, along with 
state and local assessment and 
special education personnel and 
parents. 

Step 2. Decide how each accom- 
modation will be incorporated 
into the computer-based test. 

Examine each possible accommo- 
dation in light of computer-based 
administration. Some traditional 
paper/ pencil accommodations will 
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Table 1. Steps for Developing Inclusive Computer-based Testing 



Step 1 . Assemble a group of experts to guide the transformation. 

Step 2. Decide how each accommodation will be incorporated into the computer-based test. 

Step 3. Consider each accommodation or assessment feature in light of the constructs being tested. 
Step 4. Consider the feasibility of incorporating the accommodation into computer-based tests. 

Step 5. Consider training implications for staff and students. 



no longer be needed, while others 
will become built-in features that 
are available to every test-taker. 

Step 3. Consider each accommo- 
dation or assessment feature in 
light of the constructs being 
tested. For example, what are the 
implications of the use of a screen 
reader when the construct being 
measured is reading, or the use of 
a spellcheck when achievement in 
spelling is being measured as part 
of the writing process? As the use 
of speech recognition technology 
permeates the corporate world, 
constructs that focus on writing on 
paper without the use of a dictio- 
nary or spellchecker may need to 
be reconsidered. 



Step 5. Consider training implica- 
tions for staff and students. The 
best technology will be useless if 
students or staff do not know how 
to use it. Special consideration 
needs to be given to the computer 
literacy of students and their 
experience using features like 
screen readers. Information about 
the features available on computer- 
based tests needs to be available to 
lEP teams to use in planning a 
student's instruction and assess- 
ments. Practice tests that include 
these features need to be available. 

Skipping any of these steps may 
result in the design of assessments 
that exclude large numbers of 
students. 



Step 4. Consider the feasibility of 
incorporating the accommodation 
into computer-based tests. The 
feasibility of some accommoda- 
tions may require review by 
technical advisors, or members of a 
policy/budget committee, or may 
require short-term solutions along 
with long term planning. Con- 
struct a specific plan for building 
in features that are not immedi- 
ately available, and conduct 
extensive pilot tests with a variety 
of equipment scenarios and acces- 
sibility features. 



Considerations and 




Most states have a list of possible 
or common accommodations for 
students with disabilities within 
the categories of presentation, 
response, timing/scheduling, and 
setting. Some states also list ac- 
commodations specifically de- 
signed for students with limited 
English proficiency. The list of 
considerations in Table 2 was 
generated to address the needs of 
students with a variety of accom- 
modation needs — including stu- 
dents with disabilities, students 



with limited English proficiency, 
students with both disabilities and 
limited English proficiency, and 
students who do not receive 
special services but have a variety 
of unique learning and response 
styles and needs. Here are some 
considerations for a few examples 
of specific accommodations. 

t> Large print and magnifica- 
tion (presentation) 

When type is enlarged on a screen, 
students may need to scroll back 
and forth, or up and down to read 
an entire test item. Graphics, when 
enlarged, may become very 
pixilated and difficult to view. 
Students who use hand held 
magnifiers or monocular devices 
when working on paper may not 
be able to use these devices on a 
screen because of the distortion of 
computer images. If a graphics 
user interface is used (versus text 
based), students will not have the 
option of altering print size on the 
screen. 

[> Audio presentation of in- 
structions and test items 
(presentation) 

Screen readers can present text as 
synthesized speech. The use of 
text-to-speech for test items may 
not be a viable option if the con- 
struct tested is the ability to read 
print. 
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Computer-based Tests 



Table 2. Considerations in the Transformation of Accommodations from Paper/pencil to 
Computer-based Tests 



Presentation Accommodations 



• Capacity for any student to self-select print size or magnification 

• Graphics and text-based user interfaces have different challenges 

• Scrolling issues 

• Variations in screen size 

• Effects of magnification on graphics and tables 

• Capacity for any student to self-select audio (screen reader), alternate language, or signed versions of 
instructions and test items (all students wear ear/headphones) 

• Capacity to have instructions repeated as often as student chooses 

• Variable audio speed and quality of audio presentation 

• Capacity for pop-up translation 

• Use of screen reader that converts text into synthesized speech or Braille 

• Alternative text or “alt tags” for images 

• Avoidance of complex backgrounds that interfere with readability of overlying text 

• Tactile graphics or three-dimensional models may be needed for some images 

• Capacity for multiple screen and text colors 



Response Accommodations 



• Capacity for multiple options for selecting response - mouse click, keyboard, touch screen, speech recognition, 
assistive devices to access the keyboard (e.g., mouth stick or head wand) 

• Option for paper/pencil in addition to computer (e.g., scratch paper for solving problems, drafting ideas) 

• Option for paper/pencil in place of computer (e.g., extended response items) 

• Capacity for any student to self-select spell check option 

• Capacity to disable spell check option when spelling achievement is being measured 

• Spelling implications when using speech recognition software 

• Capacity for any student to select calculator or dictionary option 



Timing/Scheduling Accommodations 



• Availability/location of computers and peripherals 

• Flexible, individualized timing 

• Capacity of network system 

• Maintaining place and saving completed responses during breaks 

• Capacity to turn off monitor/ blank screen temporarily 

• Test security 

• Capacity for self-selection of subtest order 



Setting Accommodations 



• Grouping arrangements 

• Use of earphones or headphones 

• Use of individual setting if response method distracts other students 

• Availability/comparability/location of computers and peripherals 

• Glare from windows or overhead lights 

• Adaptive furniture 

• Test security 
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^Instructions simplified/ 
clarified (presentation) 

Instructions for all students need 
clearly worded text that can be 
followed simply and intuitively, 
with a consistent navigational 
scheme between pages/items. 
Students may need an option to 
self-select alternate forms of 
instructions in written or audio 
format. 

►Write in test booklet 
(response) 

There are many options for mark- 
ing responses on computer-based 
tests that are not available on 
paper. It would still be possible for 
a student to dictate responses to a 
teacher, who would then mark 
them on the computer. The option 
of speech recognition software is 
also becoming more available. 
Speech recognition technology 
enables computers to translate 
human speech into a written 
format. Currently, speech recogni- 
tion only works for some people, 
while others, especially those who 
are not native English speakers or 
those with speech impairments, 
can be frustrated by the software's 
lack of ability to differentiate many 
of the sounds that they make. 

► Calculator (response) 

Calculator use is often allowed on 
paper/ pencil tests when arithmetic 
is not the construct being mea- 
sured. However, standardization 
of the type of calculator used has 
been very difficult and would be 
much easier if all students had the 
same online calculator to use. Use 
of an onl in e calculator is challeng- 
ing for some students, especially if 
they have not had practice with 
this tool in their daily work. 




►Breaks and multiple test 
sessions (timing/scheduling) 

Technology is required for multiple 
test sessions that would allow 
individual students to submit their 
completed responses and be able 
to log out and back on again at 
another time, starting at the place 
where they previously left off. 
Careful scheduling is needed for 
multiple test sessions to make sure 
that computers are available. Test 
security becomes an issue if stu- 
dents who have responded to the 
same test items have opportunities 
to interact with each other between 
test sessions. 



► individual or small group 
administration (setting) 

Computer-based tests create 
increased individualization for 
every student. Each student can be 
seated at a separate computer 
station wearing ear/headphones 
for audio instructions or items. 
Students using speech recognition 
systems or other distracting re- 
sponse methods need to be tested 
in individual settings. 




Summary 



With the enactment of NCLB, 
nearly all states are in the process 
of designing new assessments. As 
part of this process, several states 
are considering the use of com- 
puter-based testing, since this is 
the mode in which many students 
are already learning. Several states 
have already begun designing and 
implementing computer-based 
testing. 



Because many accessibility fea- 
tures can be built into computer- 
based tests, the validity of test 
results can be increased for many 



students, including students with 
disabilities and English language 
learners, without the addition of 
special accommodations. However, 
even though items on universally 
designed assessments are acces- 
sible for most students, there will 
still be some specialized accommo- 
dations, and computer-based 
testing must be amenable to these 
accommodations. 

Students with disabilities will be at 
a great disadvantage if paper/ 
pencil tests are simply copied on 
screen without any flexibility. Until 
the implications of the use of 
graphics versus text-based user 
interfaces are considered and 
resolved, a large number of stu- 
dents will need to continue to use 
paper/pencil tests, with a possible 
reduction in the comparability of 
results, and an increase in adminis- 
trative time and potential errors 
when paper/pencil responses are 
transferred by a test administrator 
to a computer for scoring. 
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